Freedom of Information and records management in local government: Help or hindrance?

Research into the impact of the UK Freedom of Information (FOI) Act 2000 on records management services in public authorities, especially in local government was carried out by the Department of Information Studies at UCL in 2008-2009, funded by the Arts and Humanities Research Council. The project considered the inter-relationship between records management and freedom of information, and examined the co-operation and partnerships needed in order to maximise the benefits of freedom of information. The first phase of the research was an extensive literature review, focusing on freedom of information and records management in the UK. This was followed by qualitative research using semi-structured interviews to gather rich data from council officials responsible for the provision of records management, information governance and freedom of information functions, complemented by interviews with requestors, to provide an outsider’s perspective. The article reports on the position of records management in local government prior to 2000s drawing on the literature, outlines the research findings on FOI and records management policy and practice in local government, and concludes by considering the perspective of requestors and users of the FOIA as engaged citizens. 1. Records Management in Local Authorities The practice of records management within local authorities has historically lacked uniformity from council to council.[1,2] Records management emerged as part of the county archivist’s responsibilities in the 1950s, and as a distinct activity in a local government context around the 1960s, but dedicated records management posts did not emerge until the 1980s. Even by the early 1990s, records management was not generally seen to have a strategic role in an organisation: of 33 London local authorities, only three had active records management services. [2,3] The primary obstacles to appointing records managers at that time were identified as low levels of records management expertise in the work force and the lack of motivation among senior management to introduce records management strategies. Legislation did not require culture change in records management practices before the 1990s. Although some legislation had records management implications these Acts were piecemeal, often affecting discrete functions in an authority such as access requests for medical records, social work and educational records. The Public Records Act (1958), updated in 1967, only applied to central government and excluded local government records. More specific to local government record management practices, the Local Government (Records) Act (1962) conferred limited discretionary powers for local authorities to provide certain archives services, the Local Government Act (1972) section 224 required local authorities to ‘make proper arrangements with respect to any documents that belong to or are in the custody of the council of any of their officers’, and the Access to Information (Local Government) Act (1985) included guidelines for retaining council minutes and associated background documents. These pieces of legislation provided limited incentives for authorities to manage their records within an accountable system and few legal requirements for compliance. During a 2003 UK National Archives (TNA) consultation on the possibility of new records legislation, local authorities noted the ineffectiveness of existing legislation in compelling local authorities to prioritise records management. [4] Prior to the 1990s, therefore, the development of records management in local government can be characterised as slow and failing to flourish. Yet, in the 1990s writers identify a ‘renaissance’ in local government records management, in spite of earlier failures. [3] Research by Davies and Ellis showed that within local government the number of posts advertised for qualified records managers rose from 0 in 1993 to 17 in 2001, with an increase identifiable each consecutive year. [5] Many reasons were attributed to this advance, including the changing organisational structure of authorities, increased use of ICT, improvements in records management training on university courses, and the development of performance indicators and quality initiatives. The Local Government Act (1999), for example, committed local authorities to develop long-term forward looking policies; to consult the communities they serve in order to deliver services; to deliver efficient services to high standards; and to make full use of information technology in delivering those services. The latter is often discussed with reference to the egovernment agenda, which set government departments the target of making all government services available electronically by 2005. This had clear implications for electronic document and records management systems (EDRMS) and certainly, records management was perceived by some to lie at the heart of this agenda, presupposing that there would be radical changes in local government practice. [6] Surveys of the sector in early 2000s, however, suggest that many authorities still struggled to establish records systems and the formal management of digital records in particular was still lacking. TNA, for example, noted in the report on their 2003 consultation that ‘many responses expressed concern that at present there was little or no coherent records management provision within many authorities’. [4] These weaknesses were highlighted by inquiry reports, such as the Victoria Climbié Inquiry Report, which specifically criticised poor records management practices. [7] Practitioners reported to Whitman, McLeod et al that it had taken such publicly visible inquiries to promote awareness of and real changes in record management practices in social services. [8] Again, however, rather than a full, institution-wide reform of practices, attention focused on discrete records management practices in certain departments. Digital records management was especially underdeveloped. According to Barata in 2002, local authorities ‘had yet to make any significant progress’. [9] Changes could be seen in ICT rather than in digital records management itself: E-Government activities concentrate on customer-focused services and expanding contact with users. Most efforts to date have focused on the front end, including developing Web-enabled interfaces and improving the search and retrieval aspects of customer relationship management systems. Back office services are struggling to meet demand. The broader issues of records management are often being ignored, particularly in local authorities. For some it would appear that there is a general belief that implementing a commercial document management system to underpin customer relationship services will equate with better records management. This reflects a poor understanding of the aims and benefits of records management, which is broader than document management, and the abilities of the technology employed. [9] Barata’s findings echoed the National Audit Office’s review of the first year of electronic government implementation in England. [10] They found too much emphasis on the implementation of front-end systems amongst English authorities and not enough on re-engineering local government across the entire range of processes and systems. Key questions were also raised over the sustainability of the electronic government agenda in terms of both the physical (people, finance and technical infrastructure) and non-physical resources (skill, plans and strategies) available to authorities. Benyon-Davies noted in Welsh local authorities the over-emphasis of e-government initiatives on front end systems. [11] One possible reason for this lack of attention to more holistic systems may be the perceptions of and attitudes to the kinds of changes that are necessary in organisations to comply with initiatives such as e-government. For example, immediate reactions to the demands of egovernment reforms tended to identify the need for technological change, such as providing access to information via the website. Yet, as has been argued by McLoughlin and Cornford, there are social dimensions to technological change which tend to be overlooked and neglected, including institutional context and organisation structures. [12] The diversity of structures and organisational configurations in local government suggest that changes to information systems are equally variable. These insights are also relevant to records management, as Barata suggests, since records and record systems are one product of the processes, technology and people in an organisation, and as the research results show, human factors are as important as system factors in the smooth operation of FOI. [9] The publication of the first international standard for records management in 2001, ISO15489, [13] might have been expected to address wider scale improvements to records management systems and procedures, since it provides ‘an officially endorsed benchmarking model of best professional practices for global emulation’. [14] However, an investigation by McLeod of the use of the standard in UK organisations suggested that the impact of the standard on records management practices had been limited, ‘medium to low’, and that it was the Freedom of Information Act that had the highest profile in driving change in this area and was the legislative catalyst for change in records management. [15, 16, 17, 18] The UK Freedom of Information Act 2000 came fully into force in January 2005. The UK FOI Act covers England and Wales and UK government. There is separate legislation for Scotland (the Freedom of Information (Scotland) Act 2002). The Lord Chancellor’s Code of Practice on the management of records [19, 20] published in compliance with FOIA (s 46), asserted that effective records management helps public authorities to meet their obligations under FOI. Although promoted among records management professionals and given prominence at the time of publication, in retrospect it can be argued that the Code’s high level approach, and therefore inevitably generalised guidance, left a gap which resulted in a lack of the detailed guidance local authorities needed to implement records management in practice. The Code itself is not compulsory, but it is held to be best practice. Nevertheless, the Information Commissioner’s Office (ICO) can intervene in cases where public authorities are found to be failing to meet expected standards of good practice by providing advice, carrying out assessments, issuing Practice Recommendations, and publicising positive and negative aspects of cases. [21] Research by the Constitution Unit at UCL, and by the ICO, suggested that general improvements in records management were a positive effect of FOI compliance. [22,23,24,25,26,27] Similarly positive reports of the impact of FOI on public authorities were noted for Scotland where 74% of respondents in one survey claimed that records management had improved. [28,29] The FOI Journal reported that ‘the FOIA has led to a surge in interest in the maintenance of electronic and paper records’ and that public authorities had shown ‘a marked increase in interest in records management as a general business issue’. [30] However, these results assumed that respondents were fully aware of what records management meant in practice. and there was little specific evidence for the actual ways in which records management in local authorities had improved to support these general assertions. As reported by the Constitution Unit in 2007 in a consultation for the Audit Commission, ... what we can safely say is that there are no statistics, no hard evidence, either on the improvement that FOI has actually generated in records management or in the present state generally among local authorities. [27] 2. Research project and methodology It was against this background that University College London (UCL) ran a research project over 12 months in 2008-2009, funded by the UK Arts and Humanities Research Council, which examined what the impact of the UK FOIA had been on records management services in local government. The researchers investigated how well records management services had prepared for and coped with the first three years of FOI implementation; what contribution records management services make to the ability of public authorities to comply with the FOIA; and how the user experience of FOI is affected by the management of records. The research sought to discover the impact of FOI and its link with records management from the three perspectives of records managers, institutional FOI policy managers and FOI requestors and user communities. The study concentrated on the FOI experience of local authorities. Local government was chosen partly because of weaknesses in the management of records noted above. Evidence to the Constitutional Affairs Select Committee in the House of Commons [31] reported that local authorities had less support than other sectors for FOI implementation. They ‘rely on networks and regional groups’, unlike central government they ‘do not have a clearing house’ and have ‘no hierarchy of support and advice’. In addition, ‘local authorities are still working on records management, the vast majority still do not have a corporate records management system’. Yet they deal with a high volume of FOI requests on a wide range of subjects. Following an extensive literature review [32], qualitative research methods were adopted in order to explore the issues from the perspective of the respondents based on their work context. The complexity and diversity of local government structures suggested that the organisation of FOI and records management activities in different institutional contexts would have an impact on the relationship between the two. Semi-structured interviewing was the main data collection method, which although time consuming has worked well in similar research [33]. The research focused on the south east of England, including London, which provided a study pool of 52 institutions, with examples of both small and large organizations, with and without dedicated records management professionals. Potential interview participants among local authority officers for all of these authorities were identified with the help of the London Information Rights Forum and contacted initially by email with a link to a short survey, which included a question about whether the respondent was willing to be interviewed. We also sent the same email to each authority’s general FOI request email address. All of those who responded positively were followed up. It was envisaged that ten interviews would be conducted with records managers and ten with FOI policy managers. In reality it was found that in many local authorities individuals held dual responsibilities, whilst in others the roles of FOI and records management were seen in the broader context of information governance with individuals fulfilling a wide job specification. A total of 22 interviews with 27 individuals from 19 different institutions were conducted. The transcribed interviews were uploaded into the qualitative computer software NVivo v7 to assist data analysis. One bias in the data to be acknowledged is how long those interviewed had been employed in their councils. In three cases (cases 5, 10 and 22) the interviewee had been employed in the same council for over ten years and in nine cases (cases 2, 3, 6, 11, 12, 13, 16, 19 and 21) the period of employment had been between five and ten years. These individuals had a longer-term view of before and after the implementation of the FOIA, compared with those who had been hired in preparation for FOI (cases 1, 4, 7, 15 and 17) or to cope with FOI post-2005 (cases 8, 9, 14, 18 and 20). Interviewees’ job responsibilities also affected their answers: the responses given by those with purely FOI or data protection responsibilities often displayed a more generalized notion of ‘information’, rather than of records management. The second part of the data collection concentrated on FOI requestors and users. We envisaged running focus groups with individuals from requestor groups that had made multiple requests under FOIA, since we believed that they would have an informed perspective and would have comparative experience of making requests from different councils. Inevitably their views would also be in a sense ‘expert’ and not representative of a less experienced public. From phase one of the data collection, three main groups of requestor communities were identified by our interview respondents: journalists were mentioned most frequently as users of the Act (cases 2, 6, 7, 8, 9, 10, 11, 13, 16 and 18), with political researchers, including local councillors, Members of Parliament and their researchers (cases 9, 11, 13 and 16) mentioned four times, and campaign groups four times (cases 5, 8, 12 and 17). These groups match those identified in other research [22]. Representatives of the three groups were identified, but it proved difficult to attract participants to focus groups. As a result, telephone interviews were adopted as an additional data collection strategy. Eleven interviews (9 by telephone, 2 in a focus group) were held with journalists, political researchers, campaigners, a business user and private individuals. 3. Research findings The organisation of FOI and records management functions in local authorities in our study varied considerably. In three cases among the 19 institutions in our sample, no individual had corporate records management responsibilities nor was records management a recognised corporate programme (cases 2, 20 and 22). Sometimes, whilst there was no formally acknowledged records management function, individuals looked after corporate records management alongside other duties. In those cases where records management was identified as a discrete function it was organised in a variety of ways. In nine cases, records management was linked specifically with FOI and both functions worked from the same directorate, sometimes looked after by a single individual, sometimes as part of a team. In others, the two were placed in separate directorates with varying degrees of interaction and interdependence between them. The designated parent directorate also varied: records management was most often found in an IT department (eight cases), followed by the more ‘traditional’ home within archives or libraries (although in a directorate like Adults and Communities), whilst FOI was most frequently located in a legal department. New job titles and functional descriptions were also observed, in particular the concept of information governance, which emerged in several cases (cases 1, 2, 9, 11, 19) as an umbrella for FOI-related activities. It was defined by one interviewee as: “...information governance covering the RM side and compliance and legislation and so forth.” [Interview 1] Corporate Information Manager, London Borough These different departmental contexts affect the relationship between records management and FOI. The evidence from our interviews and focus group suggests that the most effective combination appears to place FOI and records management together in the same team, with different individuals having corporate responsibility for each. [34] In this arrangement, the functions can often be situated within the broader frame of information management and governance for more dynamic, sustained, coherent approaches to information, as the overlaps between records management, knowledge and information management and information governance are considerable. Two previous articles have reported study findings relating to policy and the management of FOI and records functions in local government [34] and the implications for records management good practice [35]. Those articles reported the importance of senior management leadership both for effective FOI and for records and information management to embed the FOI/records management relationship and to provide resources. Resourcing records management continues to be a problem, especially for the appointment of dedicated staff and also for awareness raising and training of all council staff in records management practices. Our data suggests that in general our case study authorities coped with the introduction of FOI in spite of the variation of resources devoted to it, and that few felt that there had been any major compliance problems. This present article focuses in particular on the user perspective and emphasizes the importance of engaging in a dialogue with requestors to help them to understand how to use FOIA effectively as engaged citizens. 4. How is the user experience of FOI affected by the management of records? The inherent assumption in the original Code of Practice under section 46 [19] was that effective records management would be helpful in implementing the FOIA. Some councils did respond to the need to find information more efficiently in order to disclose it under FOI by organising their records better, especially in departments which were receiving a greater number of FOI requests where the argument about the link between FOI and good records management carried more weight. In general, this is good practice as set out in the Code and benefits requestors who are consequently more likely to get a timely, full and accurate reply to their request. However, a few of our respondents suggested that there is one way in which better records management can be detrimental to FOI from the requestors’ perspective, since it results in more rigorous implementation of agreed retention schedules. As councils organise their records more efficiently, they develop and implement records retention schedules more consistently. As more records are disposed of in line with retention schedules, the pool of information available to requestors becomes more restricted. Many councils reported that they were now destroying more, not in response to FOI specifically but as a result of better records management: “[departments] where there is a lot of FOI interest are far leaner than where there isn’t, because obviously when we’ve not had FOI interest we can’t use it as a driver ... three years in, and some departments or units have not had a single FOI request they’ve had to deal with. So they’re saying, well you might have got away with it ... and saying ‘this is disclosable under FOI, we’re three years in and I’ve not had a request so this is not a priority’” [Interview 5] Records Manager, County Council “It’s not uncommon that we don’t have what’s been asked for. That will become increasingly so as we manage, weed better. Unless people become experts on how the council works.” [Interview 4] Senior Information Officer, County Council This also highlights a further issue, namely the mismatch between the council’s internal structure and knowledge of records management on the one hand, and the requestor’s knowledge, making it difficult to pinpoint what the requestor is actually asking for. The requestor’s ability to frame their request accurately affects the successful outcome of the request and although public authorities are obliged to provide assistance to the requestor to frame a suitable request, they vary in their willingness to help. More than one interviewee advocated ways in which authorities could better engage with the requestor: “..actually the problem there was not entering into a fruitful discussion with the applicant. A request was made, you have no information, you have a number of opportunities to engage and you don’t do so and only when it gets to an ICO enforcement level do you magically find 1000 pages on. I am sure they could have been found earlier. Was that actually a break down in records management? Well maybe. Maybe not.” [Interview 13] Principal Information Management Officer, London Borough Requestors too emphasized that the best scenario is where there is a productive dialogue and cooperative relationship with the council. Some explained (focus group and requestors 8 and 9) that it was important for them to get the wording as specific and as concise as possible ‘because any generality they will use to refuse the request’ (focus group) instead of clarifying the request. Requestor 11 described, for instance, the problems she faced getting the questions right in the first place and she was very reliant upon the assistance of councils to identify more precisely what she was seeking, which councils did, but not all were equally helpful. Whilst this requestor was fairly positive about her interaction with FOI officers, requestor 9 expressed frustration at the difficulties in opening a dialogue with FOI officers even though he would specifically ask that they get in touch with him to discuss matters: “I get in touch and say ‘20 days have gone past’ and they just send me something rather than trying to assist me to narrow it down, or help me achieve what I am looking for because obviously you don’t know exactly what form things are kept in.” [requestor 9] Political Researcher A recurring theme was the importance of carefully framing requests for information in order to increase the chances of obtaining what is being sought. This presupposes a good degree of preparation of questions, breaking them down in several parts that can be more easily be addressed. The importance of a cooperative relationship between requestors and local authorities was emphasised. Successful FoI requests were often founded on experience in making use of the legislation. Given that the most of the individuals that we spoke to were those that had extensive experience of making FOI requests it just serves to highlight the difficulties faced by infrequent or one-off users of the Act who may be unlikely to have any prior knowledge of council structures. This is especially the case for those requestors who gather comparative data from several councils. As one said, ‘It’s awkward. In some cases different councils have different documents for the same thing’ (requestor 1). Longterm cultural change should eventually improve access: FOI is probably helping to make local authorities aware of the importance of releasing relevant information, wherever possible proactively, and to make requestors aware of the potential represented by FOI. The mismatch between the council’s internal structure and the requestor’s knowledge also highlights tensions between records management and FOI, with regard to transparency. Whilst it is the aim of the FOIA to increase transparency and accountability, the records management structures that underpin these are often not transparent. Burt and Taylor [28] noted in Scotland that FOI had become more of an internal administrative task than leverage for change, and this was echoed by our interviewees. In several comments made by the interviewees it is clear that records management has increasingly become an internally focused function. “we’re not a service delivery type department, we are kind of all the background people” [Interview 16] Feedback & Information Project Manager, London Borough Another example is case 5, in which the records manager conveyed her keenness for her function to move out of the Archives and Community Services area, as she felt that the social inclusion and community-orientated agendas of her current department were too far removed from her internal records management responsibilities. This attitude is not necessarily negative, since records management does not exist primarily for FOI but rather for the efficient functioning of the council. However, it does reflect the culture and attitude of the organisation and the role of records as evidence of business activities, activities that are in theory open and transparent: should not the records be part of that ethos? Where councils are compelled to be customer focused is in the provision of Publication Schemes, which were envisaged as a way of proactively releasing information and were thought to be the first place potential requestors would consult before proceeding with an enquiry under FOI. Each public authority has a duty to adopt and maintain a Publication Scheme and these should provide details of information which each authority publishes or intends to publish. In our research in those cases where records management and FOI functions were separated, the lead responsibility for the production of the Publication Scheme lay primarily with the FOI teams, underscoring a divide between the organisation of records from behind the scenes and the customer-facing FOI. For many the production of this document was simply an act of compliance: “When we did the publication scheme it was seen very much as an exercise in compliance. We were told we had to have one and so we did it” [Interview 5] Records Manager, County Council It is hardly surprising, therefore, given the lack of investment in these that the utility of the original publication schemes has been seen to be limited, with those produced being described as ‘hopeless’ (interview 11), ‘a waste of time’ and ‘meaningless’ (interview 14), ‘not useful’ (interview 15) and ‘a dead loss’ (interview 17). Council officers interviewed believed that the public did not consult them; one reason being that the e-government agenda had already instigated the revision of council websites to include a-z indexes and search engines. Google-type searches are more familiar ways in which the public navigate online information, rather than looking for and reading through a Publication Scheme. The requestors that we spoke to confirmed that they had consulted publication schemes in the past and were often directed to do so in response to a request, but none had found them useful. Requestors described these as ‘absolutely useless’ (focus group), ‘hasn’t been relevant’ (requestor 4), does not ‘make any difference’ (requestor 8) and ‘isn’t good enough’ (requestor 6). However, the new Publication Scheme to be introduced across councils in 2009 was seen in a more optimistic light by several of those interviewed (e.g. interviews 4, 11, 15 and 17). It remains to be seen whether the new schemes have actually improved the experience of users in identifying and requesting records. FOI is not a quick route for resource discovery for researchers. With regard to copy deadlines for instance, requestors recognised that ‘FOI based research from now on has to have no copy deadline’ (focus group), that you ‘have to think about it several weeks in advance’ (requestor 7) and that ‘you can’t really do it to a tight deadline if you are working with FOI’ (requestor 6). The approach whereby a response is supplied for the sake of complying can also be problematic if the quality of the information is not assured. For example, one of the requestors (requestor 11) reported a case where she had published figures supplied by the FOI Officer, which were in fact inaccurate, as she later discovered. The cost limit may be a barrier in the legislation to the provision of quality information. The issue of cost was mentioned in only two of our interviews with council officers (interviews 14 and 18) as one exemption they often applied. Yet it was clear from our discussions with requestors that cost had been a frequent challenge to their requests (requestors 1, 2, 9, 11 and focus group), “The usual one is that is that the paper records are going to take forever to look through them and so they excuse it on grounds of cost” [Requestor 1] Campaigner This challenge may be increasingly presented if local councils have to deal with increasing numbers of requests [36] and have inadequate records systems. “I mean we are a big council and now because there are so many requests, the extent to which I am certainly using 'going beyond the appropriate limit', it’s probably the most common response. Didn’t used to be. Quite often we use that and supply them with the information that we can do fairly readily” [Interview 14] FOIA & Data Protection Consultant, London Borough Yet, those individuals who have challenged the exemption of cost or requested substantiation for the estimated cost (requestors 1 and 9) have found that the information was supplied eventually and that the issue of cost was not as great as the council first envisaged, perhaps indicating that whoever was dealing with the request was unfamiliar with the records management systems, or that the cost of answering large or complicated requests was sometimes over-estimated. “X:...sometimes when I ask local authorities for a piece of information, some just send it back the next day and others say that it costs over £600 to produce. I have my doubts about whether that’s really true. Interviewer: So have you ever challenged that? Have you ever asked for them to substantiate? X: Yeah, I have and they tend to then produce it.” [Requestor 9] Political researcher Others chose to narrow their initial request and submit several smaller requests instead (requestors 7 and 8) or ask what is available within the limit (requestor 11). How the cost of finding information is calculated from a records management point of view remains an issue to be explored more fully. It also raises a certain dilemma in records management terms; if one council with a paperbased records management system takes longer to find something than another electronically-organised system, is the cost argument a valid challenge, if both systems function efficiently internally?